Hyperdimensional scanning transmission electron microscopy and examinations and related systems, methods, and devices

ABSTRACT

A material identification system includes one or more data interfaces configured to receive first sensor data generated by a first sensor responsive to a material sample, and receive second sensor data generated by a second sensor responsive to the material sample. The material identification system also includes one or more processors configured to generate a set of predictions of an identification of the material sample and a corresponding set of certainty information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a national phase entry under 35 U.S.C. § 371 ofInternational Patent Application PCT/US2019/057624, filed Oct. 23, 2019,designating the United States of America and published as InternationalPatent Publication WO 2020/096774 A1 on May 14, 2020, which claims thebenefit of the filing date of U.S. Provisional Patent Application Ser.No. 62/755,932, filed Nov. 5, 2018, for “SYSTEMS, DEVICES, AND METHODSFOR REALIZING HYPERDIMENSIONAL SCANNING TRANSMISSION ELECTRON MICROSCOPYAND EXAMINATIONS,” the entire disclosure of which is hereby incorporatedherein by this reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under Contract No.DE-AC07-05-ID14517 awarded by the United States Department of Energy.The government has certain rights in the invention.

TECHNICAL FIELD

Embodiments of the present disclosure relate generally to systems,devices, and methods for scanning transmission electron microscopy.

BACKGROUND

A scanning transmission electron microscope (STEM) is a type oftransmission electron microscope (TEM) in which images are formed byelectrons passing through a sufficiently thin specimen. In STEM, theelectron beam is focused to a fine spot and then scanned over the samplein a raster illumination system so that each point sample illuminatedwith the beam is parallel to the optical axis. Electrons that aretransmitted through the sample are collected by an electron detector onthe far side of the sample. The scattering of the electron beam atdifferent points on the sample depends on the sample properties, such asits atomic number and thickness.

An image is formed with the intensity of each point on the imagecorresponding to the number of electrons collected as the primary beamimpacts a corresponding point on the surface. The image contrast in aSTEM depends on detecting only electrons that are transmitted withminimum deflections (referred to as “bright field” detection) ordetecting only electrons that are scattered at an angle greater than aspecified minimum angle (referred to as “dark field” detection).Unscattered electrons are electrons that are scattered at less than apre-specified angle. If a detector were to detect all transmittedelectrons, regardless of their exit angle from the sample with respectto the electron beam axis, each pixel would have similar brightness, andthe image contrast would correspond to differences between the energy ofelectrons transmitted through different regions of the sample. Such“electron energy attenuation” contrast arises from the fact that theefficiency of STEM detectors such as solid state detectors andscintillator-photomultiplier detectors is a function of the transmittedelectron energy. However, the energy spectrum of transmitted electronsis typically relatively narrow and the corresponding electron energyattenuation contrast is weak and vastly inferior to bright and darkfield image contrast.

STEM provides the ability for improved materials research by performinga detailed structural analysis and chemistry of material specimens. Agrowing percentage of materials research now utilizes STEM-based energydispersive x-ray and electron energy loss spectroscopy to provideweighted distributions of elements at the atomic or higher size-scales.On the other hand, diffraction imaging, the only acceptedcrystallography-based technique for resolving structural-relatedinformation, is not nearly as frequently utilized due to the prohibitivespatial resolution formed using the smallest available probe formingapertures and convergence angles. By the exact nature of atomic defectsand interfaces, this mode of operation is not correlative nor compatiblewith atomic-scale STEM-based imaging and chemistry.

BRIEF SUMMARY

In some embodiments a material identification system includes one ormore data interfaces configured to: receive first sensor data generatedby a first sensor responsive to a material sample; and receive secondsensor data generated by a second sensor responsive to the materialsample. The material identification system also includes one or moreprocessors operably coupled to the one or more data interfaces. The oneor more processors are configured to: generate a first preliminary setof predictions of an identification of the material sample and acorresponding first preliminary set of certainty information responsiveto the first sensor data; generate a second preliminary set ofpredictions of the identification of the material sample and acorresponding second preliminary set of certainty information responsiveto the second sensor data; and narrow the first preliminary set ofpredictions based on the second preliminary set of predictions, thefirst preliminary set of certainty information, and the secondpreliminary second set of certainty information to generate a set ofpredictions of the identification of the material sample and acorresponding set of certainty information.

In some embodiments a method of identifying a material sample includes:generating first sensor data using a first sensor responsive to thematerial sample; generating second sensor data using a second sensorresponsive to the material sample, the second sensor different from thefirst sensor; correlating the first sensor data to material informationstored in one or more databases to generate a first preliminary set ofpredictions of an identity of the material sample; correlating thesecond sensor data to material information stored in one or moredatabases to generate a second preliminary set of predictions of theidentity of the material sample; and narrowing the first preliminary setof predictions responsive to the second preliminary set of predictionsto generate a set of predictions of the identity of the material sample.

BRIEF DESCRIPTION OF THE DRAWINGS

While the specification concludes with claims particularly pointing outand distinctly claiming what are regarded as embodiments of the presentdisclosure, various features and advantages of embodiments of thedisclosure may be more readily ascertained from the followingdescription of example embodiments of the disclosure when read inconjunction with the accompanying drawings, in which:

FIG. 1 is a block diagram of a material identification system, accordingto some embodiments;

FIG. 2 is a block diagram of control circuitry of the materialidentification system of FIG. 1 ;

FIG. 3 is a flowchart illustrating a method of identifying a materialsample, according to some embodiments;

FIG. 4 is a simplified schematic diagram of a STEM, which is an exampleof a STEM of FIG. 1 , according to some embodiments;

FIG. 5 illustrates a schematic for materials data and structure of adatabase, which is an example of databases of FIG. 1 and FIG. 2 ,according to some embodiments;

FIG. 6 is a plot of a peak distribution of family 1, according to someembodiments;

FIG. 7 is a plot of a peak distribution of family 2, according to someembodiments;

FIG. 8 is a plot of a peak distribution of family 3, according to someembodiments;

FIG. 9 is a plot of a peak distribution of family 4, according to someembodiments;

FIG. 10 is a plot of a peak distribution of family 5, according to someembodiments;

FIG. 11 is a plot of a peak distribution of family 6, according to someembodiments;

FIG. 12 is a plot of a peak distribution of family 7, according to someembodiments;

FIG. 13 is a plot illustrating a diffraction profile of an examplematerial sample, according to some embodiments;

FIG. 14 is a neural network architecture for exploitingdeep-learning-based classification for crystallographic information,according to some embodiments;

FIG. 15 is a portion of a material classification hierarchy, accordingto some embodiments;

FIG. 16 is a schematic illustration of a neural network, which is anexample of the neural networks of the control circuitry of FIG. 1 andFIG. 2 ; and

FIG. 17 illustrates confusion matrices of family level predictions,according to some embodiments.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawings in which are shown, by way of illustration, specificembodiments in which the disclosure may be practiced. The embodimentsare intended to describe aspects of the disclosure in sufficient detailto enable those skilled in the art to make, use, and otherwise practicethe invention. Furthermore, specific implementations shown and describedare only examples and should not be construed as the only way toimplement the present disclosure unless specified otherwise herein. Itwill be readily apparent to one of ordinary skill in the art that thevarious embodiments of the present disclosure may be practiced bynumerous other partitioning solutions. Other embodiments may be utilizedand changes may be made to the disclosed embodiments without departingfrom the scope of the disclosure. The following detailed description isnot to be taken in a limiting sense, and the scope of the presentdisclosure is defined only by the appended claims.

In the following description, elements, circuits, and functions may beshown in block diagram form in order not to obscure the presentdisclosure in unnecessary detail. Conversely, specific implementationsshown and described are exemplary only and should not be construed asthe only way to implement the present disclosure unless specifiedotherwise herein. Additionally, block definitions and partitioning oflogic between various blocks is exemplary of a specific implementation.It will be readily apparent to one of ordinary skill in the art that thepresent disclosure may be practiced by numerous other partitioningsolutions. For the most part, details concerning timing considerationsand the like have been omitted where such details are not necessary toobtain a complete understanding of the present disclosure and are withinthe abilities of persons of ordinary skill in the relevant art.

Those of ordinary skill in the art would understand that information andsignals may be represented using any of a variety of differenttechnologies and techniques. For example, data, instructions, commands,information, signals, bits, symbols, and chips that may be referencedthroughout the above description may be represented by voltages,currents, electromagnetic waves, magnetic fields or particles, opticalfields or particles, or any combination thereof. Some drawings mayillustrate signals as a single signal for clarity of presentation anddescription. It will be understood by a person of ordinary skill in theart that the signal may represent a bus of signals, wherein the bus mayhave a variety of bit widths, and the present disclosure may beimplemented on any number of data signals including a single datasignal.

The various illustrative logical blocks, modules, and circuits describedin connection with the embodiments disclosed herein may be implementedor performed with a general purpose processor, a special purposeprocessor, a Digital Signal Processor (DSP), an Application SpecificIntegrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) orother programmable logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general-purpose processor maybe a microprocessor, but in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Ageneral-purpose processor may be considered a special-purpose processorwhile the general-purpose processor executes instructions (e.g.,software code) stored on a computer-readable medium. A processor mayalso be implemented as a combination of computing devices, e.g., acombination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration.

Also, it is noted that embodiments may be described in terms of aprocess that may be depicted as a flowchart, a flow diagram, a structurediagram, or a block diagram. Although a flowchart may describeoperational acts as a sequential process, many of these acts can beperformed in another sequence, in parallel, or substantiallyconcurrently. In addition, the order of the acts may be re-arranged. Aprocess may correspond to a method, a function, a procedure, asubroutine, a subprogram, etc. Furthermore, the methods disclosed hereinmay be implemented in hardware, software, or both. If implemented insoftware, the functions may be stored or transmitted as one or moreinstructions or code on computer-readable media. Computer-readable mediainclude both computer storage media and communication media, includingany medium that facilitates transfer of a computer program from oneplace to another.

It should be understood that any reference to an element herein using adesignation such as “first,” “second,” and so forth, does not limit thequantity or order of those elements, unless such limitation isexplicitly stated. Rather, these designations may be used herein as aconvenient method of distinguishing between two or more elements orinstances of an element. Thus, a reference to first and second elementsdoes not mean that only two elements may be employed there or that thefirst element must precede the second element in some manner. Inaddition, unless stated otherwise, a set of elements may comprise one ormore elements.

Embodiments of the disclosure include STEM architecture and platformsthat are configured to combine imaging, spectroscopy, diffraction, andchemistry into a multidimensional dataset by operating simultaneousmodes of collection and analysis for a STEM. IN some instances thisanalysis platform may be referred to as rapid advancement ofcapability-driven examination (RACE) platform (also referred to as a“RACE engine” or simply “RACE”), a “model,” a “neural network” or merely“network,” or “deep-learning model,” and may be integrated with a STEMmicroscope. Embodiments of the disclosure include simultaneouslycollecting information over multiple modes of operation to reduce delaysin data access and extraction. The development of RACE for examiningmaterials using STEM may provide scientists and developers the abilityto perform materials discovery using the STEM. Embodiments may processand report on the atomic structure of materials using STEM, where thephase and structure of the material may otherwise be unknown or aconvolution of several phases. The typical STEM data may be extended andjoined with accompanying point-resolved chemically sensitive images andcomputational modeling, which when quantified, overlaid, and correlatedagainst one another may provide the user (e.g., a researcher and/orlicensing agency) with the phase identification and accompanying data toqualify a specific material or process.

Disclosed herein are methods of expanding the use of deep learning forcrystal structure determination based on diffraction oratomic-resolution imaging without apriori knowledge or ab-initio-basedmodeling. Of various machine learning models, such as Naïve Bayes,decision forest, and support-vector machines, convolutional neuralnetworks may produce a model with the highest accuracy. To determinecrystallography from these data types, convolutional neural network(CNN) may be trained to perform diffraction-based classification withoutthe use of any stored metadata. The CNN model may be trained on adataset including diffracted peak positions (e.g., simulated from over538,000 materials with representatives from each space group). Thegrowing potential for crystallographic structure predictions using deeplearning for high-throughput experiments is assessed herein, augmentingan ability to readily identify materials and their atomic structuresfrom as few as four Bragg peaks.

FIG. 1 is a block diagram of a material identification system 100,according to some embodiments. The material identification system 100includes a material identification apparatus 102 and one or more sensors104 (e.g., sensor 108, sensor 110, . . . , and sensor 112) operablycoupled to the material identification apparatus 102. The sensors 104are configured to generate sensor data 114 (e.g., sensor data 116 fromsensor 108, sensor data 118 from sensor 110, . . . , and sensor data 120from sensor 112) responsive to a material sample 130. The materialidentification apparatus 102 includes one or more data interfaces 106and control circuitry 200. The data interfaces 106 are configured toreceive the sensor data 114 from the sensors 104 and provide the sensordata 114 to the control circuitry 200 for processing.

The control circuitry 200 is configured to receive the sensor data 114and provide predictions of identifications of the material sample 130,as well as certainty data indicating certainty levels of thepredictions. For example, the control circuitry 200 may be configured toreceive sensor data 116 generated by sensor 108 responsive to thematerial sample 130. The control circuitry 200 may also be configured toreceive sensor data 118 generated by sensor 110 responsive to thematerial sample 130. The control circuitry 200 may further be configuredto generate a first preliminary set of predictions of an identificationof the material sample 130 and a corresponding first preliminary set ofcertainty information responsive to the sensor data 116. The controlcircuitry 200 may also be configured to generate a second preliminaryset of predictions of an identification of the material sample 130 and acorresponding second preliminary set of certainty information responsiveto the sensor data 118. The control circuitry 200 may be configured tonarrow the first preliminary set of predictions based on the secondpreliminary set of predictions, the first preliminary set of certaintyinformation, and the second preliminary set of certainty information togenerate a set of predictions of the identification of the materialsample and a corresponding set of certainty information. The set ofpredictions may be further narrowed responsive to sets of predictionsand sets of certainty information corresponding to one or moreadditional sensors of the sensors 104.

The control circuitry 200 includes one or more processors 122 operablycoupled to one or more data storage devices 124. The data storagedevices 124 include one or more databases 126 including materialinformation that is useful in identifying materials based on the sensordata 114. In some embodiments the material information includes, for avariety of different materials, diffraction data, chemistry data, imagedata, spectroscopy data, morphology data, feature size data, locationdata, other data, or combinations thereof. In some embodiments thematerial information includes peak distributions for a variety ofmaterial structures (e.g., peak distributions for families, genera,space groups, individual structures, etc.). The databases 126 includematerial identification information correlating with the sensor data 114(e.g., by comparison of peak distributions derived from the sensor data114 to stored peak distributions). By way of non-limiting example, thematerial identification information may include one or more of functionof scattering angle information, reciprocal lattice spacing information,and chemical composition information.

The data storage devices 124 also include computer-readable instructions128 stored thereon. The computer-readable instructions 128 areconfigured to instruct the processors 122 to perform operations that areuseful in identifying materials based on the sensor data 114 and theinformation in the databases 126. By way of non-limiting example, thecomputer-readable instructions 128 may be configured to instruct theprocessors 122 to perform at least a portion of the method 300 of FIG. 3(e.g., operation 306, operation 308, and operation 310). Also by way ofnon-limiting example, the computer-readable instructions 128 may beconfigured to instruct the processors 122 to perform at least a portionof functions of the control circuitry 200 discussed with reference toFIG. 2 . As other non-limiting examples the computer-readableinstructions 128 may be configured to instruct the processors 122 tooperate according to the neural network architecture 1400 of FIG. 14and/or according to the neural network 1600 of FIG. 16 . The processors122 are configured to execute the computer-readable instructions 128 toprovide predictions of materials of the material sample 130 based on thesensor data 114 and the databases 126, as will be discussed in greaterdetail herein.

In some embodiments the control circuitry 200 is configured to providethe predictions of the identity of the material sample 130 in real-timeresponsive to reception of the sensor data 114 from the sensors 104. Asused herein the term “real-time” refers to a processing duration of timethat is sufficiently short to appear close to instantaneous to humanperception. For example, substantially one second or less processingtime may be considered “real-time” for purposes of the disclosure.Accordingly, processing time of the control circuitry 200 may besufficiently fast to enable a user to modify measurement characteristics(e.g., operating settings of the sensors 104, position and/ororientation of the material sample 130 relative to the sensors 104,etc.) on the fly based on predictions provided by the control circuitry200.

In some embodiments the material identification system 100 includes aSTEM 132 including the sensors 104 and the material sample 130. The STEM132 may be configured to operate the sensors 104 to at leastsubstantially simultaneously generate the sensor data 114 to enable thecontrol circuitry 200 to process the sensor data 114 from each of thesensors 104 at least substantially simultaneously. This simultaneousprocessing of the sensor data 114 from various sensors 104 may enablethe control circuitry 200 to provide the predictions and correspondingcertainty information in real-time relative to operation of the sensors104.

In some embodiments the sensors 104 include one or more diffractionsensors (e.g., an x-ray diffraction apparatus, an electron basedscattering diffraction apparatus, a selected area electron diffractionapparatus, a high resolution atomic scale scanning transmission electronmicroscope, other diffraction sensors, or combinations thereof). In someembodiments the sensors 104 may include one or more chemistry sensors(e.g., an energy dispersive x-ray spectroscopy apparatus, an atom probetomography apparatus, a mass spectrometer, an electron energy lossspectroscopy apparatus, other chemistry sensors, or combinationsthereof). In some embodiments, the sensors 104 may include one or moreelectron imaging sensors. In some embodiments, the sensors 104 mayinclude one or more spectroscopy sensors (e.g., an x-ray spectroscopyapparatus, an electron energy loss spectroscopy apparatus, otherspectroscopy sensors, or combinations thereof). In some embodiments thesensors 104 include a diffraction sensor and one or more of a chemistrysensor, an electron imaging sensor, a spectroscopy sensor, an additionaldiffraction sensor that is different from the diffraction sensor, orcombinations thereof. In some embodiments the sensors 104 include asingle sensor including a diffraction sensor, a chemistry sensor, anelectron imaging sensor, or a spectroscopy sensor.

As the sensors 104 may include any of a variety of different types ofsensors, the sensor data 114 may include any of a variety of differenttypes of sensor data. For example, the sensor data 114 may include oneor more different types of diffraction data, one or more different typesof chemistry data, one or more different types of image data, one ormore different types of spectroscopy data, or combinations thereof. As aspecific, non-limiting example, the sensor data 114 may includediffraction data and one or more of chemistry data, image data, orspectroscopy data. As another specific, non-limiting example, the sensordata 114 may include two different types of diffraction data.

FIG. 2 is a block diagram of control circuitry 200 of the materialidentification system 100 of FIG. 1 . The control circuitry 200 includesone or more neural networks 216 operably coupled to the databases 126and a ranking stage 206. In operation the neural networks 216 areconfigured to receive the sensor data 114, provide a set of predictions212 for identification of the material sample 130 (FIG. 1 ) and a set ofcertainty information 214 indicating levels of certainty of thepredictions of the set of predictions 212. The neural networks 216 areconfigured to provide the set of predictions 212 and the set ofcertainty information 214 to the ranking stage 206.

The ranking stage 206 is configured to receive the set of predictions212 and the set of certainty information 214 and rank the set ofpredictions 212 based on the set of certainty information 214. Theprediction 208 is configured to select a prediction 208 (e.g., a highestranking prediction) from the set of predictions 212 based on theranking. The ranking stage 206 is configured to provide the prediction208 and corresponding certainty information 210 indicating a level ofcertainty of the prediction 208.

The neural networks 216 are configured to flexibly adapt to the sensordata 114. For example, the neural networks 216 may include a pluralityof parallel neural network stages 202. Each neural network stage of theparallel neural network stages 202 may be suited to process a certaintype of sensor data 114. By way of non-limiting example the parallelneural network stages 202 may include one or more diffraction stagesconfigured to process one or more types of diffraction data from one ormore different types of diffraction sensors. Also by way of non-limitingexample, the parallel neural network stages 202 may include one or moreimaging stages configured to process one or more types of image datafrom one or more different types of image sensors. As anothernon-limiting example, the parallel neural network stages 202 may includeone or more spectroscopy stages configured to process one or more typesof spectroscopy data from one or more different types of spectroscopysensors. As a further non-limiting example, the parallel neural networkstages 202 may include one or more chemistry stages configured toprocess one or more types of chemistry data from one or more differenttypes of chemistry sensors.

Based on one or more specific types of sensor data 114 received bycontrol circuitry 200, one or more corresponding neural network stagesof the parallel neural network stages 202 may be used to process thesensor data 114. By way of non-limiting example, if the sensor data 114includes diffraction data and chemistry data, the control circuitry 200may use a diffraction neural network stage and a chemistry neuralnetwork stage of the parallel neural network stages 202 to process thesensor data 114. Also by way of non-limiting example, if the sensor data114 only includes sensor data 114, the control circuitry 200 may useonly a diffraction neural network of the parallel neural network stages202 to process the sensor data 114. As a further non-limiting example,if the sensor data 114 includes diffraction data, chemistry data,spectroscopy data, and image data, the control circuitry 200 may use adiffraction neural network, a chemistry neural network, a spectroscopyneural network, and an image neural network of the parallel neuralnetwork stages 202 to process the sensor data 114.

The neural networks 216 further include a classification neural networkstage 204 configured to generate a set of predictions 212 of thematerial sample 130 and a corresponding set of certainty information 214indicating a certainty of the set of predictions 212. By way ofnon-limiting example, the classification neural network stage 204 mayproduce predictions and certainty information corresponding to each typeof sensor data (e.g., diffraction data, chemistry data, spectroscopydata, image data) of the sensor data 114, and narrow the predictions andcertainty data to the set of predictions 212 and the set of certaintyinformation 214 by processing all the different types of data as awhole.

To the extent that multiple different types of sensor data (e.g.,diffraction data and one or more of chemistry data, spectroscopy data,image data, other diffraction data, etc.) are included in the sensordata 114 the classification neural network stage 204 may generate two ormore preliminary sets of predictions and corresponding sets of certaintyinformation for the various parallel neural network stages 202. As aresult, by combining results from each of the parallel neural networkstages 202, the set of predictions 212 and the set of certaintyinformation 214 may be narrowed as compared to any of the preliminarysets of predictions and preliminary sets of certainty information.

The databases 126 include information that is helpful in identifyingvarious different materials. In some embodiments, the databases 126include crystal information files (CIFs). Information of the CIFs may begathered from open materials and crystallography databases. For example,information for the CIFs may be acquired from Materials Project, Aflow,and Open Crystallography Database (OCD). Each of the CIFs may includeinformation corresponding to a particular material. By way ofnon-limiting example, the CIFs may include crystals from all spacegroups in varying proportions. The CIFs may be used to generate astandard query language (SQL) database with relevant crystallographydata associated with computed diffraction profiles as a function ofscattering angle, reciprocal lattice spacing, and chemical compositioninformation. The examples of the databases 126 may be discussed withreference to FIG. 5 .

FIG. 3 is a flowchart illustrating a method 300 of identifying amaterial sample, according to some embodiments. In operation 302, method300 generates first sensor data using a first sensor responsive to thematerial sample. In operation 304, method 300 generates second sensordata using a second sensor responsive to the material sample, the secondsensor different from the first sensor. In operation 306, method 300correlates the first sensor data to material information stored in oneor more databases to generate a first preliminary set of predictions ofan identity of the material sample. In operation 308, method 300correlates the second sensor data to material information stored in oneor more databases to generate a second preliminary set of predictions ofthe identity of the material sample.

In some embodiments correlating sensor data to material informationstored in one or more databases (e.g., operation 306 and operation 308)includes generating a peak distribution based on the sensor data andcomparing the peak distribution to peak distributions (e.g., the peakdistributions of FIG. 6 through FIG. 12 ) stored in the databases. Insome embodiments generating a peak distribution based on the sensor dataincludes generating a binary peak distribution indicating locations ofpeaks based on one or more peak thresholds.

In operation 310, method 300 narrows the first preliminary set ofpredictions responsive to the second preliminary set of predictions togenerate a set of predictions of the identity of the material sample.

FIG. 1 is a simplified schematic diagram of a STEM 400, which is anexample of STEM 132 of FIG. 1 , according to some embodiments. The STEM400 includes a housing 402 for the internal components including one ormore electron guns 404, lenses 406, and scan coils 408. The electronguns 404 generate beams of electrons (e.g., beam of electrons 410 andbeam of electrons 412) that pass through a specimen 414. Additionaldetectors may be incorporated with the STEM 400. The detectors may beselected from the group consisting of an x-ray detector 416, an annulardark field detector 418 (e.g., a high-angle annular dark field (HAADF)detector), a camera 420 (e.g., which may take measurements via a beamsplitter 424) or other imaging device, an electron energy lossspectrometer 422, and any combination of any of the foregoing. Otherdetectors are also contemplated depending on the intended application.

One or more processors (e.g., the processors 122 of the controlcircuitry 200, one or more processors of the STEM 400 itself, otherprocessors, or combinations thereof) may be configured to performvarious operations in conjunction with the STEM 400. For example, theprocessors may be configured to perform and/or control operation,acquisition, and processing of simultaneous images, spectra, anddiffraction patterns collected on the STEM 400. An additional RACEplatform may bridge multiple gaps in nanoscale analysis that currentlyexist with conventional STEMs.

In particular, embodiments of the disclosure may control easilyconfigurable multipoint resolved two-dimensional (2D) mapping in theSTEM 400 where for each pixel (e.g., within 1 nm×1 nm or smaller) thereis an associated indexable 2D diffraction pattern, a quantifiable energydispersive x-ray or electron energy loss spectrum, and intensitycontrast value provided by annular dark or bright field STEM detectors.Based on these inputs, the RACE platform may be configured to indexmaterials to within the pixel resolution and quantify element specificchemistry. In addition, the RACE platform may be configured to combinecrystallographic information and chemical composition to establish rapidmaterials discovery on the STEM based, at least in part, on thecollected STEM data leveraging known techniques in big data mining andrecovery.

Establishing the foundation for a materials discovery platform throughthe joint application of point resolved nanocrystallography andchemistry may ultimately provide a framework to rapidly advance theability to identify nanochemistry, phases, trace impurities, andnanostructures in seemingly bulk materials from a variety ofapplications. Intuitively, the combination with computational tools suchas density functional theory and atomic-based dynamics simulations onthe effects of irradiation or the accumulation of fission products areeminent examples of what potential high-impact scientific discoveriesand validations can be correlated in detail with ACE. As a result, STEMsmay be improved to accelerate the scientific pace and reduce the needfor expert-level familiarity with microscopy, crystallography, andproviding the future with a foundation for a materials and scientificdiscovery for extended technical impacts on biology, chemistry, physics,material science, and energy sciences.

Materials research may be improved by enabling and combining multiplemodes of operation inside a STEM to form a single all-encompassingmaterials dataset. Embodiments of the disclosure include enabling allmodes available in the microscope to form a multimodal (e.g.,multidimensional) dataset where combining chemistry and structure isnatural. For example, the following operations may be performed andexecuted by the RACE platform integrated with the STEM 400.

By way of non-limiting example, one or more processors may perform ascan process, which may be set up including defining parameters such asdwell time, pixel size, frame time, readout frequency, and scanfrequency. An initial STEM image may be collected for the sample. A beampath may be calculated, including computing an efficient pixel beam pathbased on the initial STEM image. A multi-dimensional imaging may be setup, including implementing a chosen beam path and input beam positions.A chosen path and positions may be implemented by a field-programmablegate array that controls the operation of the scan coils 408. Feedbackcontrol for the RACE platform may be performed, such as predicting therate of drift and correcting the scan area.

Also by way of non-limiting example, one or more processors may enabledetectors (e.g., x-ray detector 416, annular dark field detector 418,camera 420, electron energy loss spectrometer 422). Data may be capturedby the different detectors. Such data may be combined into an N^(th)dimensional data set containing data from the different detectors at agiven point in time. The data may be combined and quantified, such asthrough image reconstruction, chemical & structure quantification ofEDS, EELS, and diffraction data. The RACE platform may further performcombinatorial comparisons within the different information of the dataset. The RACE platform may perform visualization of the data. Forexample, the various results may be evaluated using parallel processingand visualized on a display device. The display device may present aresearcher with a user-friendly interface that displays datainterpretation, data trends, and reports, such as regarding the originor identity of a specific material and/or its properties.

As a result, the data extraction and access of cutting edge researchtools may enable the combination of multiple modes of operation toextend attainable information without compromising time. The ability torender a single multi-dimensional dataset, including imaging,spectroscopy, diffraction, and chemistry may be an improvement overother microscopy applications. This improvement may be due to the RACEplatform simultaneously acquiring images, spectroscopy, diffractioninformation, and/or chemistry information to enable a completeunderstanding of intricate atomic scale interfaces, defects, impurities,and generalized features. Such information is often responsible formeasureable differences in electronic, mechanical, thermal, and physicalproperties of materials.

In nuclear materials research, nuclear fuels and advanced metallurgicalalloys may benefit from a combined multimodal approach, where using aprecision controlled atomic-size (e.g., Angstrom-size) electron beamwith minimal distortion may be utilized to report radiation inducedfeatures, such as stacking faults, dislocations, phases, and diffusionto the atomic scale without compromising information. For example,embodiments of the disclosure may be applied to understand the role ofmaterial defects, interfaces, and phases associated with advancedcladdings (e.g., oxidedispersion alloys and MAX phases), fuel (e.g.,U-Mo, UO2 dispersion fuel), and spent nuclear fuel and waste products(e.g., 137CsCl, 90SrF) to inform multiple material campaign efforts,including those related to the nuclear fuel cycle, advanced fusionreactors, and accident tolerant reactor technologies.

As another example, samples (e.g., xenon-irradiated samples such as inthe superlattice of uranium-molybdenum fuel or as solid stateprecipitates in high temperature aluminum) may be used by the STEM todevelop routines for not only reliably differentiating chemistry at thenanoscale, but also calculating combinatorial differences in atomicordering that are easily predicted thru density functional theorycalculations and molecular dynamics simulations.

By enabling simultaneous imaging, spectroscopy, and diffraction modes inthe STEM 400, structural and chemical information is naturally mergedinto a single multi-dimensional dataset. Based on the natural formationof this multi-dimensional dataset, different types of data (e.g.,structure, chemistry, morphology feature size, location, etc.) may bequantified, overlaid, joined, and compared against one another withoutcomplications of scale, size, or field of view. A user-friendlymultiplatform software may be configured to combine and accelerate datacollection, quantification, and provides a foundation for materialsexploration, validation, and discovery. Embodiments of the disclosuremay be applied to various industries including nuclear, materials,microscopy (e.g., microscopy examinations), and other scientificindustries using analytical instruments such as STEM.

The RACE platform architecture is configured to handle and processimages, spectra, and diffraction patterns collected on the STEM 400.Based on results a modeling module may validate results. For example,the RACE engine (e.g., the control circuitry 200 of FIG. 1 and FIG. 2 )may be configured to perform various operations such as advancedmathematics and statistics, peak and function fitting, componentmapping, machine learning, and automation. In addition, the RACE enginemay receive information from the various modes of operation that mayoperate substantially simultaneously. These modes may include materialsmodeling configured to perform phase field modeling, first principlesmodeling, atomistic modeling, and advanced crystallography. Adiffraction analysis may include phase and material identification,orientation mapping, and texture analysis. An imaging analysis mayinclude image sizing, and feature segmentation. A spectroscopy analysismay include energy dispersive x-ray spectroscopy, and an electron energyloss spectroscopy.

In some embodiments, the engine and modes may be configured asopen-access extensions to an open-platform for a STEM, whereas otherembodiments may be configured as a standalone product (e.g., in Java,Labview, Matlab/Simulink, Python-based programming suites, etc.). Insome embodiments, the RACE platform may be integrated into existingSTEMs (e.g., by retrofit). As a result, the addition of specificphysical and signal hardware may be added separately to the STEM, suchas smaller size condenser apertures, field programmable gate arrays, andmultifunction I/O modules to interface with the beam position andsimultaneous signals. In some embodiments, the RACE platform may beintegrated within STEMs at the outset during initial design andmanufacture such that additional components may simply be incorporatedat the outset prior to being set up with a customer.

Implementation of a RACE platform architecture using machine learningmay, in one embodiment, employ a neural network for determination ofcrystal structure based on diffraction and/or atomic-resolution imaging.

In one example the STEM 400 and RACE platform may be used to study azirconium hydride alloy including heterogeneous samples containinghydride platelets that may be studied in detail with imaging,diffraction, spectroscopy, chemistry information, and validated by phasefield modeling.

In one example the STEM 400 and RACE platform may be used to studybimodal samples that have been Xenon (Xe) irradiated includinguranium-molybdenum (U-Mo) alloys and room temperature aluminum (Al) tocombine structural and chemical information at the atomic level toperform phase identification and mapping.

In one example the STEM 400 and RACE platform may be used to studyinterplanetary space dust where, based on the information gained, highlyimpactful insights on the origins of the early solar system may beprovided. The collected interplanetary space dust and meteorites may befrom the NASA Stardust mission. These early universe samples are highlyscientifically valued for their origins and may provide the ultimatetest in the ability to report on the elemental speciation and presenceof unknown phases caused by background cosmic radiation on the originsof early life on Earth.

In one example various data sets from the STEM 400 may be analyzed bythe RACE engine and examples of different deliverables may be displayedto the user. For example, the RACE engine may receive a samplemorphology dataset of oxide dispersion strengthened alloy with Ti and Crcontaining precipitates, and perform advanced image segmentation todifferentiate particles and grains, and particle counting, to generatean image showing segmented grains based on the STEM image. In anotherexample, the RACE engine may receive a material structure dataset of abright field image taken from a grain interior. The RACE engine mayperform particle size counting, structural mapping, and a strainanalysis to generate decomposed structural FFTs of the Fe-CR matrix andthe particles.

In another example, the RACE engine may receive a material chemistrydataset of a STEM-based electron energy lost spectroscopy for variouselements (e.g., Ti, Fe, O, Cr). The RACE engine may perform advancedspectral analysis applied to electron energy loss spectroscopy,quantified maps, uncertainty analysis, principal component analysis,phase decomposition, and visualization to generate relative atomiccomposition maps and an accompanying high resolution image. In anotherexample, the RACE engine may receive a 3D imaging dataset such as anx-ray based tomographic image of a graphite block. The RACE engine mayperform segmentation of features and 3D visualization to generate anadvanced 3D composition and structural map of the granulates andparticles.

FIG. 5 illustrates a schematic for materials data and structure of adatabase 500, which is an example of databases 126 of FIG. 1 and FIG. 2, according to some embodiments. FIG. 5 illustrates a structure 502, arelative composition pie diagram 504, and a data allocation diagram 506of the database 500. In the database 500, a pre-simulation datasetincludes 572,000 CIFs. Additional CIFs (e.g., from the inorganic crystalstructure database (ICSD)) for underrepresented space groups wereprovided to provide further examples. The CIFs were then used togenerate an SQL database with relevant crystallography data associatedwith computed diffraction profiles as a function of scattering angle,reciprocal lattice spacing, and chemical composition information.

The structure 502 of the database 500 indicates the structure used tocreate and store the training set. For example, each entry of thedatabase 500 includes information indicating a name, a chemical formula,a Hall group, a space group, a genera, a family, a relative genera, arelative species, an entry identification, a diffraction link, achemistry link, and an HKL link. The diffraction link informationincludes information regarding intensity, theta binned, and D-spacebinned. The chemistry information includes information regarding atomicpercent, count, element, formula identification, and entryidentification. The HKL information includes information regardingmultiplicity and HKL index.

The relative composition pie diagram 504 illustrates an allocation ofthe database 500 to each crystallographic family, genera and species.For example, each entry of the structure 502 may be directed to a generaof the relative composition pie diagram 504. The genera of FIG. 16include a monoclinic 508, a triclinic 510, an orthorhombic 512, a cubic514, a tetragonal 516, a trigonal 518, and a hexagonal 520 genera. Therelative composition pie diagram 504 illustrates an abundance (e.g., themonoclinic 508 genera, the triclinic 510 genera) and a scarcity (e.g.,the hexagonal 520 genera, the trigonal 518 genera) of certain genera inthe database 500. An abundance or scarcity of certain families may alsobe present.

The data allocation diagram 506 illustrates data allocation during thedifferent folds (e.g., fold 1 through fold N) during training andtesting for cross-validation. Fold 1 through fold N−1 may be associatedwith augmentation and training. Fold N may be associated with test.

Fingerprints for any material based on crystal geometry, underlyingatomic coordination, and occupancy may be encoded into a diffractionprofile. Under the influence of an impinging x-ray and destructiveinterference neutron or electron results in a series of peaks inscattered intensity where there is destructive interference, forming atwo-dimensional diffraction pattern. Pending on scattering geometry onespecific pattern is generated per sample orientation, where a singlecrystal for one orientation will show a series of identifiable peaks inreciprocal space. Filling in all reciprocals requires a highlypolycrystalline sample or alternatively a sample that is processedthrough all sample orientations and diffracting conditions completingthe Ewald sphere where all diffraction peaks can be identified andreadily indexed. For the purposes of classification, all orientationsand identifiable peaks are input into the model. Chemistry may be inputinto the model as an additional descriptor. Presence of specificelements and composition may be implemented as additional search termsfor structure or may be used to further parse potential structures fromclassified models.

The CIFs may be checked for consistency and proper formatting. CIFs thatare missing structures, chemical formulas, or whose symmetry operationsare inconsistent with their space group may be removed from the trainingset. The removal may be performed consistent with established sets ofcrystallographic rules and classes. CIFs missing one essential field,such as structure, may often also be missing other fields. Afterchecking the CIFs for formatting, diffraction profiles may be simulated(e.g., using Python Materials Genomics (PyMatgen)). The spectra may beconverted into a single feature vector by thresholding peaks based ontheir relative intensity compared to the highest peak in each signal.Profiles that contain no peaks in a range between 0.50 and 6 Angstromsmay also be removed from the training set. Remaining spectra may then bestored in a SQL database with labels for family, genera, species andchemistry. In the example of the database 500 the cleaned training setincluded 431,000 spectra and associated chemical metadata, includingcomposition.

Simplifying the representation of diffraction and chemistry allows for abroader set of data acquisition methods. Whether the input (e.g., sensordata 114 of FIG. 1 and FIG. 2 ) is a Fourier transformed high resolutionatomic scale image, or diffraction profile acquired using electrons,neutrons, or X-rays the relevant atomic scattering peaks are positionedwith respect to their crystallographic scattering position in reciprocalspace. A classification model therefore that considers peak positionalone in reciprocal space is impervious to changes in technique.Training data was therefore built from CIFs and simulated a wealth ofavailable features to train models. For example, a minimalistrepresentation may represent features that are expected to be present inthe widest range of acquisition methods. For diffraction, the profilemay be reduced to a vector of peak locations. Peaks may be representedin the vector as 0 if no peak was detected and 1 if a peak was detected.The peaks may be divided into 900 bins uniformly partitioning the rangefrom 0.5 to 6 Angstroms in reciprocal d spacing. This range mayaccommodate a wide selection of microscopes and hardware. The number ofbins may have sufficient but not excessive granularity based on theunderlying resolution of 0.1 Angstroms, which is below the experimentaluncertainty of 0.3 Angstroms in reciprocal spacing. By imposing fewerrequirements and assumptions on the model inputs a generalized model maybe created. Chemistry may be input as a feature vector containingelement listing and computed atomic compositions.

In addition to the simulated diffraction profiles, a set of augmentationoperations may be defined on the data set to be used during training tofurther bolster the training data including relative peak assignmentuncertainty of 0.3 Angstrom, in reciprocal space. The uncertainty may bechosen based on the level of common refinement methods and sources.

Neural networks may rely on larger training sets than other machinelearning algorithms. In order to address both the scarcity and imbalanceof rarer classes of materials in the databases (e.g., databases 126,database 500), a set of functions that would mimic data collected withinan experimental setting may be defined. For example, two augmentationsfor the diffraction input and one for the chemistry may be defined. Thefunctions may be chosen to replicate experimental variations that areplausible across all data acquisition modalities.

Diffraction augmentation accounts for variations within cameracalibration and peak localization methods. Peak positions may be shiftedby a number of bins drawn from a normal distribution centered at zerowith a variance of 1.5 bins, where the width of a bin is equal to 0.0061/Angstroms. The range of possible shifts may be chosen to account fordifferences in binning method, centering of experimental data, anddispersion variation over the entire input profile.

For atomic percentage a composition may be allowed to change by up to 5atomic % (at. %) or 5 parts per million (ppm) to mimic the experimentaluncertainty amongst common methodologies. Methods for chemicalcomposition analysis of materials include energy dispersive x-rayspectroscopy (EDS), atom probe tomography (APT), mass spectrometry (MS),and electron energy loss spectroscopy (EELS). Pending quantifiedstandards that calibrate the results of these techniques there is upperuncertainty bound of 5 at. %, where this value has been implemented intothe model to cover a significant range and higher ablation value. Withhigher certainty the statistics lends itself to improved classification,where potential for higher background is captured for higher ablation.

A robust processing pipeline may be developed for different collectionmodalities of diffraction to create the feature vector for model input.Two-dimensional diffraction data may be azimuthally integrated to createa profile in pixel space used alongside calibration settings todetermine the d-spacing of the peaks. To detect peak positions inreciprocal spacing, profiles may be processed through a max votingalgorithm. The voting algorithm may or may not use a backgroundsubtraction to fit peaks. The voting algorithm may instead utilizes amax polling variational profile to define a rising feature as a peak.The detected peak locations may be binned and cataloged by position.Chemistry data may be implemented as a simple binary vector to capturethe presence of elements in a material and if available, a vector ofatomic percentage.

In another example, approximately 650,000 individual structures werescreened for duplicates or other potential errors. A weighting schemawas applied for each class and assigned to address the occurrence ofoverly represented crystal types, noting substantial imbalances amongstpopulated crystal families, groups, and space groups ranging from136,534 to less than 1,000. The aggregate accuracies and populationstatistics for each level of the hierarchy are reported in Table 1.

TABLE 1 Accuracy (%) Population Triclinic 91.04  105,200 pedial N/A16,740 pinacoidal N/A 88,460 Monoclinic 86.73% 217,156 sphenoidal 86.50%21,705 domatic 74.95% 14,997 prismatic 90.14% 180,454 Orthorhombic75.75% 104,526 rhombic-disphenoidal 77.18% 22,182 rhombic-pyramidal92.32% 19,722 rhombic -dipyramidal 67.42% 62,622 Tetragonal 84.81%40,770 tetragonal-pyramidal 65.80% 1,081 tetragonal-disphenoidal 76.46%1,437 tetragonal-dipyramidal 96.23% 5,112 tetragonal-trapezohedral82.99% 2,373 ditetragonal-pyramidal 88.16% 1,450tetragonal-scalenohedral 84.22% 4,309 ditetragonal-dipyramidal 81.14%25,008 Trigonal 82.78% 31,252 trigonal-pyramidal 81.48% 3,499rhombohedral 90.46% 6,017 trigonal-trapezohedral 94.13% 2,321ditrigonal-pyramidal 82.90% 5,831 ditrigonal-scalenohedral 89.43% 13,584Hexagonal 86.07% 24,147 hexagonal-pyramidal 88.17% 1,828trigonal-dipyramidal 90.14% 453 hexagonal-dipyramidal N/A 2,100hexagonal-trapezohedral 99.24% 930 dihexagonal-pyramidal 92.01% 2,969ditrigonal-dipyramidal 95.36% 3,230 dihexagonal-dipyramidal 93.64%12,637 Cubic 95.47% 48,289 tetartoidal 96.75% 1,842 diploidal 90.56%1,389 gyroidal 93.68% 737 hextetrahedral 97.14% 6,475 hexoctahedral87.29% 37,846

Crystal files were labeled by family, genera, and species. Thecorresponding label was used for different levels of the hierarchy totrain and evaluate models. Table 1 summarizes the population andaccuracy overall in seven crystal classes and 32-point symmetry groups.Each file generated a distinct diffraction profile as a function of thecorresponding Bragg angle, utilizing crystal structures. Interplanard-spacings were generated at each level of the hierarchy to train themodel. The resolution in the individual binary signal, including anormalized vector of intensity against the Bragg angle, was set to 0.5°.

Out of 571,340 total structures stored in the database 500 at the familylevel, over 136,534 randomly chosen structures were evaluated at thefamily level and the genera level to evaluate the model (e.g., neuralnetworks 216 of FIG. 2 ). When compared at the genera level, cubic andorthorhombic confusion matrices illustrate the hierarchy andclassification scheme issues of imbalance in the data despite theweighting during training. This example highlights the classificationhierarchy analogous to nested network architecture capable of predictingstructure at the family, genera, and space group levels.

As previously mentioned, initially there were approximately 650,000structure-based files that were cleaned and simulated. The files werescreened for formatting errors, missing essential information, andsimulation errors. The files were used to simulate diffraction profileswhile they were checked against the CIF file to ensure propersimulation. The profiles were further refined to a binary signal of peakpositions. The simulated signals contained several prominent peaks anddozens of lesser peaks that may not be present in all experimentalsettings and differed between electron, X-ray, and neutron experiments.In the case of electron microscopy, the intensity does not necessarilyscale against known structure factors and is strongly affected bymaterial texturing. For full-scan X-ray and neutron data in which theintensities scale against known structure factors, an applied thresholdwas used to remove peaks below the signal-to-noise ratio to seed theprediction with fewer peaks leads to predicting crystal with a highdegree of accuracy (e.g., well above 80%). The prominent diffractionpeaks are the most reliable indicator of the structure. The previouscrystallography analysis tools further corroborate the model. Based on abinary representation of the data as a function of peak position, thehierarchical model was trained on signals for each family and genera,removing peak intensity as a variable, which simplifies therepresentation.

Moving to the binary representation of peak positions eliminated theintensity of the peaks and allowed the models to be applied to severaldiffraction-based modalities. A simulated diffraction profile includeslattice spacings as a function of either Angstroms or integrated Bragg.Due to constraints on the number of learnable parameters, the Braggangle resolution was 0.5°. Based on a survey of peaks at 0.5°, it is areasonable resolution for classification in the cases of 60 to 300 kVelectron beams, typically operating voltages of modern electronmicroscopes.

If space group classifications may be learned from peak locations alone,aggregate signals for each family, genera, and species may be summedacross all members and quantitatively compared. The aggregate peakdistribution signals for triclinic, monoclinic, orthorhombic, and cubicfamilies are plotted against two theta values in FIG. 6 through FIG. 12. The peak distributions for each family, genera, and species may bestored in the database 500 for comparison to material sample peakdistributions generated from sensor data (e.g., the sensor data 118) toidentify a material sample (e.g., material sample 130). For example, apeak distribution for the material sample may be generated based ondiffraction data, image data, chemistry data, spectroscopy data, orcombinations thereof. The peak distribution for the material sample maybe compared against peak distributions (e.g., those of FIG. 6 throughFIG. 12 and others) to identify a family, genera, and species of thematerial sample. As a specific, non-linear example, a least squaresregression may be used to compare the material sample peak distributionto the various peak distributions stored in the database 500 todetermine which stored peak distributions are closest to the materialsample peak distribution at various levels (e.g., the family, genera,and species levels).

FIG. 6 through FIG. 12 are plots of peak distributions over genera orfamilies at each level to uniquely differentiate notable features. Atthe family level of the hierarchy, significant overlaps in distributionsand similarities among genera caused predictions among families to beunreliable. Once the family of the crystal is determined, predictionaccuracies rise into the 80-98% range.

In some embodiments the database 500 is configured to store a peakdistribution for each structure stored in the database (e.g., 571,340peak distribution signals for 571,340 structures). The peak distributionsignals are not uniformly distributed amongst all classes at any levelof the hierarchy. There is a strong preference for higher-symmetrystructures within the seven crystal families at the genera level, butthere does not appear to be a preference at the space group level. Themodel was trained on the materials contained within each database. Thedisparity in membership between classes introduces challenges intotraining deep-learning models.

Datasets that are highly imbalanced are susceptible to mode collapse. Insuch a case, predicting the most common class yields a high accuracywithout discriminating between materials. To counteract the imbalancepresent in the crystallographic data, a weighting schema may be imposedin training. During training, the relative presence of classificationwas used to compute a weight that would be applied during the losscalculation. The weight may be defined as the ratio of the total numberof structures per space group over all space groups. The same weightingschema may be performed at the point group level.

In cases where the dataset is relatively balanced, the number ofexamples in each class may be within the same order of magnitude; theweights may be similar enough that they do not significantly bias themodel. When a dataset is highly imbalanced, the number of examples ineach class may differ by more than an order of magnitude. This schemamay penalize the model for incorrectly predicting a rarer space groupmore harshly than it rewards the model for correctly predicting a commonspace group. Weighting the rare classes to be more important to themodel during training had an ameliorating effect on the data imbalancebut does not eliminate it. Models trained without this weighting schemasuffered mode collapse and would not predict rare classes. To accountfor and further mitigate mode collapse from data imbalance, models weretrained on top-one accuracy but evaluated on top-two accuracy.Misclassifications are predominantly to the common class, and top-twoaccuracy of the ranked predictions allows the model, in many cases, tocorrect for misclassifications due to imbalances in the data. Therelative score from the output of the SoftMax layer determines the rankorder. The confidence in the ranked prediction is based on the modelaccuracy during testing, not the relative score.

Initial test models showed that attempting to classify directly tospecies produced models with poor accuracy and compounding effects frommode collapse. Utilizing a hierarchical model that assumes family, thenpredicts genera, then species leads to higher accuracy at each step.Even with cascading error the method is substantially more accurate.Comparing the confusion matrices from each family highlights a highlevel of accuracy.

FIG. 6 is a plot 600 of a peak distribution 602 of family 1, accordingto some embodiments. In family 1 membership was 105,200 members. Anaverage number of peaks of the peak distributions of the members offamily 1 was about 3.79 peaks.

FIG. 7 is a plot 700 of a peak distribution 702 of family 2, accordingto some embodiments. In family 2 membership was 217,156 members. Anaverage number of peaks of the peak distributions of the members offamily 2 was about 3.37 peaks.

FIG. 8 is a plot 800 of a peak distribution 802 of family 3, accordingto some embodiments. In family 3 membership was 104,526 members. Anaverage number of peaks of the peak distributions of the members offamily 3 was about 3.15 peaks.

FIG. 9 is a plot 900 of a peak distribution 902 of family 4, accordingto some embodiments. In family 4 membership was 40,770 members. Anaverage number of peaks of the peak distributions of the members offamily 4 was about 2.38 peaks.

FIG. 10 is a plot 1000 of a peak distribution 1002 of family 5,according to some embodiments. In family 5 membership was 31,252members. An average number of peaks of the peak distributions of themembers of family 5 was about 2.45 peaks.

FIG. 11 is a plot 1100 of a peak distribution 1102 of family 6,according to some embodiments. In family 6 membership was 24,147members. An average number of peaks of the peak distributions of themembers of family 6 was about 2.36 peaks.

FIG. 12 is a plot 1200 of a peak distribution 1202 of family 7,according to some embodiments. In family 7 membership was 48,289members. An average number of peaks of the peak distributions of themembers of family 7 was about 1.74 peaks.

FIG. 13 is a plot 1300 illustrating a diffraction profile 1302 of anexample material sample, according to some embodiments. The diffractionprofile 1302 is an intensity plotted against a Bragg angle. The examplematerial sample included crystalline strontium titanate (SrTiO3) islandson a face-centered cubic structured magnesium oxide (MgO) substrate. Aneural network architecture (e.g., the neural networks 216 of FIG. 2 )and workflow was validated based on high-resolution STEM imaging andelectron diffraction from the example material sample. Utilizinghigh-resolution capabilities of an aberration-corrected STEM withsub-Angstrom resolution, an atomically resolved high-angle annular darkfield image of the individual Sr, Ti, and O atoms was taken orientedalong the [100] zone axes by atomic species. A Fast-Fouriertransformation (FFT) of the atomically resolved image taken along thesame orientation, along with this preferred crystallographic direction,was computed. Based on the FFT, the diffraction profile 1302 is atwo-dimensional azimuthal integration of the pattern transformed into aone-dimensional profile. The pattern and diffraction profile 1302provide the structural classification details for classifying andpredicting the structure using a deep-learning model approach.

As previously mentioned, the diffraction profile 1302 was taken based onSTEM image data. A similar profile as the diffraction profile 1302 maybe obtained simultaneously based on diffraction data. Any material in ascanning transmission electron microscope (STEM), in this case,crystalline SrTiO3 (STO) islands distributed on a rock salt MgOsubstrate, can be simultaneously imaged with high resolution atomic masscontrast STEM imaging and decoupled with a selective area, Fast-Fouriertransform (FFT) to reveal the material's structural details. Based oneither electron diffraction or FFT of an atomic image (or both), atwo-dimensional azimuthal integration translates this information into arelevant one-dimensional diffraction intensity profile (e.g., thediffraction profile 1302) from which the relative peak positions inreciprocal space can be indexed. Seeding the prediction ofcrystallography is a hierarchical classification that may utilize aone-dimensional convolution neural network model replicated at eachlayer from family to space group, forming a nested architecture (see theneural network architecture 1400 of FIG. 14 ). Based on the derived peakpositions in the azimuthal integration profile, the prediction on SrTiO3is reported in Table 2 below. Table 2 compares from higher to lowersymmetry materials, a prediction for various materials.

TABLE 2 Material Expectation 1^(st) Prediction 2^(nd) Prediction 3^(rd)Prediction CeO₂ Cubic Fm3m 225 (87.1%) 219 (5.6%) 221 (3.6%) No. 225C-graphene Hexagonal 194 (90.1%) 173 (2.5%) 191 (0.2%) P6₃/mmc No. 194Bi_(1.15)Sb_(0.71)Te_(0.85)Se_(2.29) Trigonal R3m 166 (26.1%)  163(25.7%)  148 (3.48%) (BSTS) No. 166 A-phase U Orthorhombic  74 (34.7%) 19 (33.7%)  63 (15.9%) Cmcm No. 63

FIG. 14 is a neural network architecture 1400 for exploitingdeep-learning-based classification for crystallographic information,according to some embodiments. The neural network architecture 1400builds and trains on public and established materials databases,including the Open Crystallography Database (OCD), Materials ProjectDatabase (MPD), American Mineralogist Crystallographic Databases (AMCD),and the Inorganic Crystal Structure Database (ICSD). The neural networkarchitecture 1400 is a one-dimensional convolutional neural network(CNN) architecture used to train and evaluate a hierarchical trainingdataset including 571,340 individual crystals divided amongst sevenfamilies, 32 genera, and 230 individual crystallographic space groups.At each level of the hierarchy, a neural network was trained to form anested hierarchy for classification as shown in FIG. 14 . Each CNNincludes six convolutional blocks 1436 before three dense layers (e.g.,dense layer 1428, dense layer 1430, and dense layer 1432) and a SoftMax1434 for classification. Convolutional blocks 1436 are formed from aconvolutional layer (e.g., convolutional layer 1402, convolutional layer1404, convolutional layer 1406, convolutional layer 1408, convolutionallayer 1410, and convolutional layer 1412), a max pooling layer (e.g.,max pooling layer 1414, max pooling layer 1416, max pooling layer 1418,max pooling layer 1420, max pooling layer 1422, max pooling layer 1424),and an activation layer. The convolutional layers have a kernel size oftwo and start with 180 channels narrowing to 45 channels over the sixblocks. Starting in the fourth block (e.g., convolutional layer 1408 andmax pooling layer 1422), dropout is applied after the pooling layer.Dropout starts at 0.1 (at max pooling layer 1420) and scales up to 0.2(at max pooling layer 1422) and 0.3 (at max pooling layer 1424) in thefifth and sixth blocks, respectively.

Due to a lack of grand canonical examples or a human baseline, theneural network architecture 1400 was benchmarked by comparing it toother machine-learning algorithms. Convolutional neural networksoutperformed other machine learning methods, including decision forests,support vector machines, and the Naive Bayes model. In certainsituations, random forests appear to outperform convolutional neuralnetworks. Upon delving into the random forest models, it is revealedthat random forest models are subject to mode collapse. Despite themodels having a high accuracy (of eighty-plus percent), the model hasn'tlearned to distinguish classes. It predicts the class that comprises 80%of the data every time. Convolutional neural networks are susceptible tomode collapse as well, which occurs most prominently when classifying acrystal into a family.

Optimizing the deep learning model involved tuning varying architecturaland training hyperparameters. The model architecture (e.g., the neuralnetworks 216 of FIG. 2 ) was tested at varying depths, numbers ofparameters, layer combinations, and dropout rates. The final modelarchitecture selected includes a flatten layer 1426 and six blocks ofconvolution (the convolutional blocks 1436). Max pooling (max poolinglayer 1414, max pooling layer 1416, max pooling layer 1418, max poolinglayer 1420, max pooling layer 1422, and max pooling layer 1424) anddropout were selected based on performance, the number of trainableparameters, and preservation of spatial information. The dense layers(dense layer 1428, dense layer 1430, and dense layer 1432) were tuned.The three dense layers before classification were optimal for producingaccuracy.

During hyperparameter optimization, a batch size of 1,000 was chosen inconjunction with weighting by the class occurrence to increase theprevalence of rarer classes in the gradient of each batch. The number ofpeaks used to classify structure was the hyperparameter had asignificant effect on the prediction accuracy. The number of peaksincluded is determined by a threshold of peak intensity applied to thesimulated patterns. Stricter thresholds, 80-90% intensity of the mostprominent peak, produced signals with fewer peaks. Relaxed thresholdsbelow 50% produced signals with increasingly many peaks. Using athreshold stricter than 90% of max peak intensity almost universallyeliminates all but the maximum peak.

Optimizing structural parameters involved introducing dropout in earlylayers of the network, which prevented the neural network from learningfrom sparse signals. Dropout may be implemented to prevent overfittingof data by ignoring portions of noisy signals. A heavily processedbinary peak signal that only contains peak locations for three to sixpeaks may be utilized in training. The binary peak signal may begenerated from the azimuthal integration of an FFT or SADP as arotationally invariant profile where individual peak locations can beidentified. A number of peak finding techniques may be used. By way ofnon-limiting example, a peak finding tool may include a moving windowtype max voting peak finder that populates a binary signal sampled atless than 0.03 Angstroms in real d-spacing. Such sparse vectorrepresentations of the data, including dropout early in the model, mayeliminate the signal before propagating to learnable features. This mayresult in poorly operating models. Instead, dropout may be introducedgradually starting in the third convolution block (e.g., convolutionallayer 1406 and max pooling layer 1418), starting at 0.10 and increasingto 0.30, in the last convolutional layers (convolutional layer 1408,convolutional layer 1410, and convolutional layer 1412) to preventoverfitting. The six blocks of convolution (convolutional blocks 1436),max pooling, and dropout condense the signal but maintain the spatialrelevance of the original data.

To supplement the data and provide a more robust training regimen forthe limited data, cross-validation may be used instead of splitting thedata into single training, testing, and validation sets. Forcross-validation the data may be split into ten folds. For each fold amodel may be trained on the other nine folds. The resulting models maybe compared with each other to test for overfitting and generalization.This process may be repeated for each level of the hierarchy. Validationmay be performed using experimental data that was not part of thetraining process for the models.

The number of peaks present were compared with the measured accuraciesfor each family based on the accompanying confusion matrices. Theconfusion matrices were organized across seven crystal families (e.g.,corresponding to the peak distributions of FIG. 6 through FIG. 12 ) atthe genera level, which constitutes the class hierarchy, followed by thenumber of peaks used. An identical effect was observed at the spacegroup level as well. Comparisons to other machine-learning methods,including decision forests, support vector machines, and the Naïve Bayesmodel are reported in Table 3.

TABLE 3 Model Family Class Space Group Random 14.3% 10-33%  4-33% CNN 80% 75-95% 65-99% Random Forest  51% 30-60% 52-84% Naïve Bayes 16.4% 5-15%  2-11% SVM N/A 38-47% 20-57%

The training and tuning of models may be performed on a high-performancecomputing system such as an Nvidia DGX-1 system. However, to ensure thatthe model is usable in a setting where high-performance computingresources are not available, speed benchmarking was done on anentry-level single graphical processing unit (GPU) desktop. In otherwords, the deep-learning model was evaluated for real-time analysis on asingle GPU desktop machine to evaluate the efficiency, sensitivity, andcomputing necessity for performing augmented crystallographicdetermination in an accessible manner. The single GPU machine used atruntime had a GTX 1050ti 3 GB graphics card and a 3.2 GHz i7 quad-coreprocessor. Running on this single GPU desktop, the model classifiedbatches of 1,000 profiles loaded in sequence at a rate of 2,600 to 3,500predictions per second. Conversely, when the same profiles are loadedindividually, the model classifies significantly more slowly at a rateof 29 predictions per second. Classification speeds are the same forpredictions across families, genera, or space groups. Predicting spacegroups for 48,289 observations consisting of a single family took 13.2seconds, which is equivalent to 3,525 predictions per second.Subsequently, the ability to classify structure from diffraction insub-second times allows for significant acceleration in the acquisitionand prediction of at least two orders of magnitude based on the currentability of experts to predict and augment the analysis without previousknowledge.

Based on the input from the atomic resolution SrTiO3 STEM image, theranked order of predictions made from the FFT-image-based profiles is:225 (Fm3m), 221 (Pm3m), 205 (Pa3). Upon validation with the knowncrystal structure, the crystal was determined as structured as spacegroup number 221, Pm3m, subsequently oriented along the [100] zone axescontaining the [200] and [110] family of crystallographic reflections.Utilizing the data workflow and pipeline provides the generalizedframework for classifying all known materials.

Alongside predictive accuracy, performance on single GPU machines duringmodel tuning and design may also be considered. Though trained on an HPCmachine, the model was designed to be deployable on any readilyavailable single GPU machine or an entry-level cloud-based end point.Though the model is capable of classifying at a rate exceeding 3,500predictions per second for a large batch of preprocessed diffractionsignals, this does not consider the time necessary to processdiffraction patterns into the appropriate input form. The model iscapable of handling large backlogs of data with this predictive speed,but a more realistic workflow for newly generated data would be runningsmall batches or sequentially predicting for each observation. Whenanalyzing each diffraction image through the full pipeline, includingthe azimuthal integration and peak-finding algorithms, there may be aslowdown in predictive speed. This may be the case in a current workflowfrom raw to the processed peak position. At 22 predictions per second,real-time analysis of a live camera feed may be achieved.

FIG. 15 is a portion of a material classification hierarchy 1500,according to some embodiments. The material classification hierarchy1500 includes a genera of cubic 1512, and point groups includingtetrahedral 1502, hextetrahedral 1504, diploidal 1506, gyroidal 1508,and hexoctahedral 1510. The tetrahedral 1502 point group includes spacegroup index numbers 195-199. The hextetrahedral 1504 point groupincludes space group index numbers 200-206. The diploidal 1506 pointgroup includes space group index numbers 207-214. The gyroidal 1508point group includes space group index numbers 215-220. Thehexoctahedral 1510 point group includes space group index numbers221-230. The neural network architecture 1400 of FIG. 14 is configuredto map a diffraction profile (e.g., the diffraction profile 1302 of FIG.13 ) to one or more of the space group indexes of the materialclassification hierarchy 1500.

A generalized workflow and accompanying network for crystallographicmaterials such as STO at nanometer scale provides the capability toderive crystallographic structure from high-resolution images. Atomicresolution images translate into crystallographic patterns, and anazimuthal integration of the crystallographic pattern resolves theindividual interatomic d-spacings and accompanying Bragg angles forsubsequent crystal prediction and refinement. The input to the modelseeds a deep-learning model for nested prediction, utilizing over571,340 crystals to provide a capability for deriving crystal structure.Top-two accuracy may be used to discriminate between classes.

After training and tuning the models on a synthetic dataset the modelswere validated on experimental data. Several well-known materials withknown crystallographic structures representative of ongoing materialsresearch programs were selected. The sparse sampling of materialsenabled validation of the processing pipeline and hierarchical model.These materials range from cubic to orthorhombic. The materials wereeach representatives of a crystal family. Higher-symmetry cubic crystalswere started with from low to high resolution for nanocrystalline CeO2.

For all of these predictions, the model displays a level of sensitivityto the number of peaks used for classification. In several cases, morethan four peaks were detected. Using more than four peaks can presentsome ambiguity as to which peaks should be used for classificationwithout apriori knowledge. In these cases, simple heuristics may be usedto narrow down the selection of peaks. In the case of electrondiffraction, peaks below the resolution limit of 0.5 Angstroms and aboveeight Angstroms (e.g., heavily diffraction limited) may be ignored.Despite this range, this is a robust model to perform generalizedcrystallographic classification. For example, in the case of BSTS, twodifferent, valid sets of peaks may be fed into the model, generating twodifferent sets of predictions. Future explorations of these differentpossible detected peak combinations may lead to further improvements onthe model and removal of ambiguities in the predictions.

FIG. 16 is a schematic illustration of a neural network 1600, which isan example of the neural networks 216 of the control circuitry 200 ofFIG. 1 and FIG. 2 . With respect to FIG. 16 and FIG. 17 , developmentand demonstration of deep learning models for materials classificationand discovery from separate or combined prospective of materialstructure and chemistry will be discussed. Modular neural networksprovide a flexible framework to build multimodal models from. With anaverage accuracy above 85% at each level of the hierarchy a deeplearning model may predict a space group of an unknown crystal structurewithout any a priori information. By providing a ranked list of possiblespace groups and chemistries, the deep learning-based model and workflowrepresents a milestone toward fully automated materials applications,where readily identifying materials and their behavior may be provided.

With respect to FIG. 16 and FIG. 17 , disclosed is a modular neuralnetwork architecture and a simplified and generalizable representationof crystallography and chemistry to classify crystal structure. Thegeneralized representation of diffraction and chemistry data allows forflexibility for users to include in their workflow. The servicedemonstrates a workflow and analysis tool for high throughputcharacterization. By reducing complexity and reliance for familiarity,the disclosed embodiments lend themselves to additional discoveryopportunities for under-represented and poorly understood materials.Embodiments disclosed herein may also include a method for creating adeep learning service and provide a demonstration of the increased speedof an automated workflow. Embodiments disclosed herein may alleviaterepetitive time-consuming tasks and provide a simple workflow tomitigate multidimensional challenges associated with higher throughputmethods, which rely on expert knowledge of material structure andchemistry.

Various types of machine learning algorithms are available for use inthe neural networks 216. For example, machine learning algorithmsincluding random forests, naïve Bayes, and support vector machines(SVMs) may be compared with artificial neural networks to determinewhich algorithms would be best suited for the task of structuralcharacterization. Training may be performed using a processor (e.g., theprocessors 122 of FIG. 1 , such as graphical processing units (GPUs)).Neural networks may have many positive properties and higher predictivecapabilities for this task than other machine learning algorithms. Theavailability of large datasets and augmentation methods may make itpossible to train neural networks. Neural networks represent a class oflearning algorithms. Convolutional neural networks (CNNs) (e.g.,multiple pairs of convolution layers and max pooling layers) of adiffraction stage architecture 1610 may be used to capture thediffraction inputs (e.g., diffraction data 1666 of the sensor data 114of FIG. 1 and FIG. 2 ) due to their spatial component. For example, thediffraction stage architecture 1610 of FIG. 16 includes four such pairsincluding convolution layer 1622 and max pooling layer 1624, convolutionlayer 1626 and max pooling layer 1628, convolution layer 1630 and maxpooling layer 1632, and convolution layer 1634 and max pooling layer1636. Each pair of layers may also include a normalization layer (notshown) and an activation layer (not shown) in addition to theconvolution layer and the max pooling layer. Diffraction stages (e.g.,diffraction stage 1602) are composed of sequential convolution blockpairs ending with a flatten operation (e.g., flatten 1620).

A chemistry stage architecture 1612 may be used to capture chemistryinputs (e.g., chemistry data 1668 and chemistry data 1670 of the sensordata 114 of FIG. 1 and FIG. 2 ) to produce a chemistry output 1650(e.g., corresponding to chemistry output 1616 and chemistry output 1618of chemistry stage 1604 and chemistry stage 1606, respectively). Thechemistry stage architecture 1612 may include a series of dense layers(e.g., dense layer 1638, dense layer 1640, dense layer 1642, dense layer1644, dense layer 1646, and dense layer 1648) of a chemistry stagearchitecture 1612. Chemistry stages (e.g., chemistry stage 1604) includesequential dense blocks ending with a single dense layer to create asimplified representation of chemistry. Each of the dense layersincludes three layers, a dense layer, a normalization layer, and anactivation layer, and are designed to find relationships betweennon-spatially ordered variables.

As previously indicated, the neural network 1600 combines structure(e.g., diffraction) and chemistry. Accordingly, the neural network 1600includes a diffraction stage 1602 configured to process the diffractiondata 1666 and two chemistry stages, chemistry stage 1604 and chemistrystage 1606, configured to process the chemistry data 1668 and chemistrydata 1670, respectively. In some embodiments, one or more of thediffraction data 1666, the chemistry data 1668, and the chemistry data1670 may include diffractograms. The diffraction stage 1602, thechemistry stage 1604, and the chemistry stage 1606 may be examples ofthe parallel neural network stages 202 of FIG. 2 . The neural network1600 also includes a classification stage 1608 configured to combineflatten 1620 (e.g., an output of the diffraction stage 1602) with achemistry output 1616 of the chemistry stage 1604 and a chemistry output1618 of the chemistry stage 1606. The chemistry output 1616 and thechemistry output 1618 may be concatenated, which are in turnconcatenated with the flatten 1620, and provided to the classificationstage 1608.

The classification stage 1608 has a classification stage architecture1614 including various dense layers (e.g., dense layer 1652, dense layer1654, dense layer 1656, dense layer 1658, dense layer 1660, and denselayer 1662) and ending with a SoftMax layer 1664.

Directly classifying space group is complicated by over representedclasses. A hierarchical approach, such as that of the neural network1600, decomposes the difficult classification problem into smaller andmanageable tasks. The material classification includes three stagesalong phylogenic lines: classification of crystal family, symmetrygroups, and space groups. In keeping with the phylogenic schema, thesehierarchical levels may be referred to as family, genera, and species.At each level of the hierarchy a new model may be trained. An ensembleof models to predict family, and then within each family, models may betrained to predict genera, and lastly for each genera, models may betrained to differentiate species. Due to the large branching nature ofthe schema, models may not be trained end to end, but instead useprevious predictions to determine the proceeding model to use next inthe next sequence of processing.

Combining diffraction data 1666 and chemistry data (chemistry data 1668and chemistry data 1670), the neural network 1600 may learn fromdistinct inputs. The architecture was created using four stages: threedesigned to learn from the input data types and one to perform a task,in this case classification. The modular design was created to testvarious hypothesis and allow for flexibility in training and retrainingportions of the network, without challenges to retrain an entirenetwork, but as submodules. The modular architecture may be extended andretrained in parts to incorporate additional datatypes should other databecome available.

The diffraction stage 1602, which includes a series of convolutionallayers with max pooling (e.g., diffraction stage architecture 1610) maybe configured to capture the spatial component of the signal. Lacking aspatial component, chemistry is captured by stacked dense layers (e.g.,chemistry stage architecture 1612). During training each layer may befollowed by a normalization layer. The outputs of each stage (e.g.,diffraction stage 1602, chemistry stage 1604, and chemistry stage 1606)may be concatenated and used as feature vectors for the classificationstage 1608, which includes the classification stage architecture 1614including a series of stacked dense layers ending with a SoftMax layer1664 for classification.

Stages at different levels of the hierarchy are based on the samearchitecture. Due to the large number of hyperparameters to test, theoptimal parameters, which were found at the family level stage, may beapplied to all genera and species level stages. The genera and specieslevel stages may use different final SoftMax layers than the familystage to accommodate for the varying number of classes. Ablation studiesmay show this optimizing of each stage separately using a recurrentneural network architecture.

As a specific, non-limiting example, a network for classifying familymay include a diffraction module, a chemistry module and aclassification module. The diffraction module may include four stackedblocks of convolution, pooling, normalization and activation layers. Theinitial convolutional layer may include a 3×3 kernels with an outputtensor of 1×40×900. After pooling the output may be the batch size1×40×450. Repeating the process of convolution and pooling three timesyields a final output shape of 1×40×112, which is then flattened into a1×4032 tensor to be concatenated with the output of the chemistrymodule. The chemistry module includes four stacked blocks of dense,normalization, and activation layers. The initial chemistry input is a1×118 tensor including the atomic composition of the elements present inthe material sample (e.g., the material sample 130 of FIG. 1 ). Thedense layer includes 20 nodes and subsequent layers include 15, 11, and8 nodes, respectively.

The outputs of the two modules may then be concatenated and put into theclassification module dense layers. The classification module had fourblocks of dense, normalization, and activation layers. Dense layers had500, 250 and then C nodes, respectively, where C is the number ofclasses at the stage in the hierarchy. For example, if theclassification module was for families, C would be seven. The last layerof the classification module is a SoftMax layer.

Due to the imbalance in membership at all levels of the hierarchy aleave one out cross validation split may be used to generate trainingand validation data instead of a single training, validation, andtesting set. A single balanced testing set would have either been smallor used all the examples of the rare classes. Instead, the data may besplit into five folds, trained on four, and tested against the remainingportion. Models trained on each fold combination may be aggregated andcompared to determine how much the model is overfitting and to see ifgeneralization occurred.

In order to further address the imbalance between classes, a weightingmay be applied to the loss function at training to incentivize correctlypredicting a rare class. In addition, data augmentation may be used tofurther bolster rare classes during training to generate more samples.

Starting with a modular architecture presents some challenges whenoptimizing the complete architecture because it introduces severalpotential axes for tunable hyperparameters. In some embodiments, anarchitecture includes three modules that can include a variable numberof layers and types of connections. Initial attempts to determine whichhyperparameters to hold fixed and which to include in the optimizationin one off model comparisons may be used to see if changing a specificparameter (e.g., layer depth, stride, and kernel size) yields noticeablechanges to model performance. Due to the size of the search space,initial comparisons may be between partially trained models to increasethe cycle time on iterations. Parameters of the augmentation functionsmay be held constant across all levels within the hierarchy.

To determine which portions of the model contribute most to predictiveaccuracy, an ablation study may be performed at each level of thehierarchy that compares variations of the model. Augmentations producedifferent effects at each level of classification. In order to elucidatewhich portions of the model are most impactful for classification,variations on the model may be trained with the same hyperparameters. Atthe family level, models that include only the diffraction module, onlythe chemistry module, permutations on augmentation and withoutnormalization may be tested. Due to the combinatorial nature of possiblecombinations of attributes for an ablation study, for the genera andspecies level the ablation studies may start with diffraction only andthen added additional features.

To determine what role chemistry and diffraction play on classification,versions of the model that only incorporate one modality may be trainedfor comparison. A model containing only chemistry may have limitedpredictive power. Within the cubic class the model may perform at about98% accuracy and may have a significantly higher chance of accuracy thanrandom selection. A model trained only based on diffraction performswell across all families with an average accuracy of about ˜88%, withthe largest drop in accuracy between monoclinic and triclinic families.These classes represent those with minimal symmetric operations,resulting in primitive representation on atomic arrangement ofmaterials.

FIG. 17 illustrates confusion matrices 1700 of family level predictions,according to some embodiments. The confusion matrices 1700 include achemistry only confusion matrix 1702, a diffraction only confusionmatrix 1704, a diffraction with wider bins confusion matrix 1706, adiffraction only without normalization during training confusion matrix1708, a diffraction only with normalization confusion matrix 1710, adiffraction and chemistry confusion matrix 1712, a diffraction andchemistry with diffraction augmentation confusion matrix 1714, and adiffraction and chemistry with combined augmentations confusion matrix1716. In the confusion matrices 1700, predicted and expected familyclassification, where predicted is the vertical and expected is thehorizontal starting with triclinic (1), monoclinic (2), orthorhombic(3), tetragonal (4), trigonal (5), hexagonal (6), and cubic (7). Thechemistry only confusion matrix 1702 is trained on chemistry only, thediffraction only confusion matrix 1704 is trained on diffraction only,the diffraction with wider bins confusion matrix 1706 is trained ondiffraction with wider bins, the diffraction only without normalizationduring training confusion matrix 1708 is trained on diffraction onlywithout normalization during training, the diffraction only withnormalization confusion matrix 1710 is trained on diffraction only withnormalization, the diffraction and chemistry confusion matrix 1712 istrained on diffraction and chemistry, the diffraction and chemistry withdiffraction augmentation confusion matrix 1714 is trained on diffractionand chemistry with diffraction augmentation, and the diffraction andchemistry with combined augmentations confusion matrix 1716 is trainedon diffraction and chemistry with combined augmentations. Values in theconfusion matrices 1700 are in decimals (0.01 being equal to 1%).

The diffraction with wider bins confusion matrix 1706 shows the effectthe number of bins has on the model's ability to predict. The model maybe trained using a reduced feature vector with 180 features instead of900, where parameters including number of layers, stride, kernel size,normalization may remain the same. The model accuracy suffersnoticeably, dropping an average of −30% accuracy across all families andmode collapse is observed in the band of misclassifications surroundingorthorhombic (3).

Diffraction only without normalization during training confusion matrix1708, diffraction only with normalization confusion matrix 1710,diffraction and chemistry confusion matrix 1712, and diffraction andchemistry with diffraction augmentation confusion matrix 1714 are modelsutilizing both diffraction and chemistry data. These models had keyfeatures of chemistry augmentation, diffraction augmentation andnormalization removed to evaluate those features' effectiveness. Thereare marginal improvements from adding in diffraction augmentation, ˜1-2%improvement in orthorhombic—cubic, but a decrease in performance ofmonoclinic of ˜5%. Allowing the atomic percentage to vary by a margin of5 at. % decreases performance at lower symmetry without noticeableincrease in accuracy at higher symmetries when predicting family. Thisbehavior suggests materials are not organized over crystal family.Within crystal family, however, considering chemistry improves theclassification by as much as 9% as reported below in Table 4.

TABLE 4 Diffrac- Diffrac- tion and Only Diffrac- tion ChemistryChemistry Diffrac- tion with Augmen- Augmen- Augmen- tion Chemistrytation tation tation Triclinic 0.975 +0.001 −0.005 −0.002 −0.006Monoclinic 0.849 +0.007 −0.052 −0.009 −0.075 Orthorhombic 0.801 −0.011−0.022 +0.007 −0.022 Tetrahedral 0.805 −0.009 −0.007 −0.013 −0.025Trigonal 0.884 +0.005 +0.013 +0.012 +0.009 Hexagonal 0.889 −0.001 −0.008−0.006 +0.006 Cubic 0.728 +0.091 −0.051 +0.081 −0.042

Table 4 illustrates results of a family to genera ablation study.Numbers reported in Table 4 are averages across all genera presentwithin the family. Individual genera may perform better than the averagefor the family. Within each family common genera have higher accuracythan rarer genera. Values in Table 4 are reported in decimals (e.g.,0.01=1%).

With general trends apparent at the family level, a reduced set of modelvariations were used in the ablation study of the genera levelclassification. The set of five variations tested were: diffraction,with chemistry and no augmentation, with chemistry and chemistryaugmentation, with chemistry and diffraction augmentation, withchemistry and both kinds of augmentation. Table 4 shows the averagechanges across all genera within each family as different features inthe model are included. Adding in chemistry improved the predictivepower of models across all crystal families except hexagonal. Thelargest improvement in accuracy was distinguishing between cubic generawith an average 9% increase in accuracy, but contains significantlyhigher improvements for less common genera.

Chemistry augmentation had a positive effect on more balanced datasets,with an increase of an average of 2-7% accuracy for most classes.Orthorhombic and Tetragonal crystal families having only chemistryaugmentation during training decreased correct classifications ofuncommon genera by ˜10-15%. Diffraction augmentations had a positiveeffect for predicting genera within the trigonal and hexagonal families,with an average increase of 2-4%. For the other crystal familieschemistry augmentation had a negative effect, lowering accuracies by8-12%. Within the cubic family distinguishing peaks are tightlyclustered distributions with less variance, allowing diffraction peakposition to shift by more than 0.02 Angstroms or 3 bins obscurescritical information and produce worse models more prone to modecollapse. At lower symmetry there is an imbalance within the data wherethe peak distributions have higher variance consistent with a reducednumber of geometrical operations for these more primitive classes.

Combined augmentations produced more consistent models for predictinggenera within Tetragonal, Trigonal, and Hexagonal crystal families withlower variance in performance between the models trained on differentfolds and better accuracy for rare and uncommon classes, decreases inaccuracy for common classes. The tradeoff between higher accuracy forcommon classes or better predictive power for rare classes represents aninteresting choice when considering how the model may be used. Thevariance in performance from the augmentation functions appears to be afunction of data balance and prevalence as well as symmetry within acrystal family.

Table 5 below illustrates a summary of results of a genera to speciesablation study. Change in accuracy may be a change in raw accuracy.

TABLE 5 3 4 5 6 7 8 9 10 11 12 13 14 15 Baseline Raw 91.2 92.8 82.4 87.473.7 72.6 33.2 92.7 75.5 73.6 88.8 85.2 85.5 Baseline Scale 92.0 93.179.7 65.5 72.0 64.1 38.1 78.3 67.7 67.9 74.0 77.7 81.6 With Chemistry+0.4 −0.2 +0.5 −1.0 −1.6 +7.7 −3.0 −4.8 +6.5 −2.4 +0.2 −0.9 +0.7Diffraction Aug −0.6 +1.3 −1.6 −8.0 +4.0 +5.5 +1.7 −1.2 +0.7 +1.1 +2.2+2.8 +5.1 All Augments −0.9 +1.2 −3.0 −7.7 +3.6 +7.6 −5.1 −0.7 +1.4 −1.9−1.8 +2.3 +5.0 16 17 18 20 23 24 25 26 27 28 29 30 31 32 Baseline Raw92.9 95.7 83.5 94.3 98.9 63.6 98.7 96.2 98.1 86.1 17.5 62.4 95.9 83.8Baseline Scale 79.4 92.2 76.1 89.6 80.8 68.1 90.9 94.8 95.8 66.5 17.581.3 88.3 65.2 With Chemistry −1.1 −0.2 −0.4 −0.6 −0.1 +5.4 −0.4 +0.9−0.3 −0.7 −2.9 +3.3 +0.2 +7.3 Diffraction Aug +0.7 +0.7 −3.4 +1.8 −0.5+11.5 +0.2 +1.1 −0.4 +0.6 +3.6 +3.8 −2.0 +1.5 All Augments −0.1 +0.7−4.2 +2.0 −0.6 +6.5 +0.2 +1.4 +0.0 +5.4 −1.3 −0.7 −2.1 +0.9

At the species level imbalances between classes creates a much morenoticeable effect with mode collapse affecting several genera within theorthorhombic and cubic crystal families, with individual speciescomprising greater than 90% of the population in their genera. Twodifferent accuracies are reported in Table 5 to highlight the disparitybetween common and rare species. Raw accuracy is the percentage ofcorrectly classified profiles across all species, and scaled accuracy isthe average accuracy of each species within the genera. An occurrence ofraw accuracy being higher than scaled accuracy is a symptom of imbalancewhere the trained model becomes preferential to common species due toprevalence in the training set. Even outside of the extreme cases,imbalance between classes increased when going from the genera level tothe species level. Models with just diffraction, combined diffractionand chemistry, and combined data with augmentation were compared for theablation study and results are captured within Table 5. Chemistry had amore pronounced effect, improving performance predicting species withinall genera between 10-35%. Adding in both chemistry and diffractionaugmentations led to mode collapse and worse models in a majority ofgenera.

Despite significant ability to classify materials, it should be notedthat information contained in the training data is not uniformlydistributed across all space group, crystal families, or materialclasses. It is unknown if the abundance of crystals in the commonclasses is representative of the true distribution of materials orsampling bias that is a product of past research efforts beingconcentrated on specific materials. The imbalance between space groupswithin the data set may prove to be one of the greatest challenges inproducing good models. For the purposes of training and evaluating adeep learning model for material classification, a hierarchical set ofmodels may be trained in light of the imbalances including chemistry,whereas if information becomes available the models can be furtherimproved.

Examples

A non-exhaustive, non-limiting list of example embodiments follows. Notall of the example embodiments listed below are individually indicatedas being combinable with all others of the example embodiments listedbelow and embodiments discussed above. It is intended, however, thatthese example embodiments are combinable with all other exampleembodiments and embodiments discussed above unless it would be apparentto one of ordinary skill in the art that the embodiments are notcombinable.

Example 1: A system, comprising: a scanning electron transmissionmicroscope (STEM); and an analysis platform integrated with the STEM andconfigured to combine imaging, spectroscopy, and diffraction into amulti-dimensional dataset by operating simultaneous modes of collectionand analysis for the STEM.

Example 2: The system of Example 1, wherein the analysis platform isfurther configured to resolve two-dimensional (2D) mapping in the STEMfor each pixel measured within a predetermined range.

Example 3: The system of Example 2, wherein the predetermined range isless than about 1 nm×1 nm.

Example 4: The system of Example 1, wherein the multi-dimensionaldataset includes at least an associated indexable two dimensionaldiffraction pattern, a quantifiable energy dispersive x-ray or electronenergy loss spectrum, and an intensity value associated with atomiccontrast high-angular annular dark field STEM imaging.

Example 5: The system of Example 1, wherein the modes include one ormore of a materials modeling mode, a diffraction analysis mode, animaging analysis mode, and a spectroscopy analysis mode.

Example 6: A method for operating a scanning transmission electronmicroscope (STEM), the method comprising simultaneously acquiringimages, spectroscopy, and diffraction information with STEM images andstoring the simultaneously acquired data as a multi-dimensional datasetfor analysis and display by an interface coupled to the STEM.

Example 7: A system at least substantially as shown in the drawingfigures and described in the specification.

Example 8: A device at least substantially as shown in the drawingfigures and described in the specification.

Example 9: A method at least substantially as shown in the drawingfigures and described in the specification.

While the present disclosure has been described herein with respect tocertain illustrated embodiments, those of ordinary skill in the art willrecognize and appreciate that it is not so limited. Rather, manyadditions, deletions, and modifications to the illustrated embodimentsmay be made without departing from the scope of the present disclosureas hereinafter claimed, including legal equivalents thereof. Inaddition, features from one embodiment may be combined with features ofanother embodiment while still being encompassed within the scope of thepresent disclosure. Further, embodiments of the disclosure have utilitywith different and various detector types and configurations.

What is claimed is:
 1. A material identification system, comprising: oneor more data interfaces configured to: receive first sensor datagenerated by a first sensor responsive to a material sample; and receivesecond sensor data generated by a second sensor responsive to thematerial sample; and one or more processors operably coupled to the oneor more data interfaces, the one or more processors configured to applythe first sensor data and the second sensor data to one or more neuralnetworks to: generate a first preliminary set of predictions of anidentification of the material sample and a corresponding firstpreliminary set of certainty information responsive to the first sensordata; generate a second preliminary set of predictions of theidentification of the material sample and a corresponding secondpreliminary set of certainty information responsive to the second sensordata; and narrow the first preliminary set of predictions based on thesecond preliminary set of predictions, the first preliminary set ofcertainty information, and the second preliminary second set ofcertainty information to generate a set of predictions of theidentification of the material sample and a corresponding set ofcertainty information.
 2. The material identification system of claim 1,wherein the first sensor comprises a diffraction sensor including atleast one of an x-ray diffraction apparatus, an electron basedscattering diffraction apparatus, a selected area electron diffractionapparatus, or a high resolution atomic scale scanning transmissionelectron microscope.
 3. The material identification system of claim 2,wherein the second sensor comprises a chemistry sensor including atleast one of an energy dispersive x-ray spectroscopy apparatus, an atomprobe tomography apparatus, a mass spectrometer, and an electron energyloss spectroscopy apparatus.
 4. The material identification system ofclaim 2, wherein the second sensor comprises an electron imaging sensor.5. The material identification system of claim 2, wherein the secondsensor comprises a spectroscopy sensor including at least one of anx-ray spectroscopy apparatus and an electron energy loss spectroscopyapparatus.
 6. The material identification system of claim 2, wherein thesecond sensor comprises another diffraction sensor that is differentfrom the diffraction sensor of the first sensor.
 7. The materialidentification system of claim 1, wherein the first sensor datacomprises diffraction data.
 8. The material identification system ofclaim 7, wherein the second sensor data comprises at least one ofchemistry data, image data, and spectroscopy data.
 9. The materialidentification system of claim 7, wherein the second sensor datacomprises diffraction data that is different from the diffraction dataof the first sensor data.
 10. The material identification system ofclaim 1, wherein the one or more processors are configured to rank theset of predictions from most certain to least certain based on thecorresponding set of certainty information and generate a predictionincluding a highest ranked prediction of the set of predictions topredict the identification of the material sample.
 11. The materialidentification system of claim 1, wherein the one or more datainterfaces is further configured to receive third sensor data generatedby a third sensor responsive to the material sample, wherein the one ormore processors are further configured to apply the third sensor data tothe one or more neural networks to: generate a third preliminary set ofpredictions of the identification of the material sample and acorresponding third preliminary set of certainty information responsiveto the third sensor data; and narrow the first preliminary set ofpredictions based on the third preliminary set of predictions and thethird preliminary set of certainty information in addition to the secondpreliminary set of predictions, the first preliminary set of certaintyinformation, and the second preliminary second set of certaintyinformation to generate the set of predictions of the identification ofthe material sample and the corresponding set of certainty information.12. The material identification system of claim 1, wherein the one ormore processors are configured to generate the set of predictions of theidentification of the material sample in real time responsive to receiptof the first sensor data and the second sensor data.
 13. The materialidentification system of claim 1, further comprising one or more datastorage devices including one or more databases stored thereon, the oneor more databases including material identification informationcorrelating with the first sensor data and the second sensor data,wherein the one or more processors are configured to generate the firstpreliminary set of predictions and the second preliminary set ofpredictions based, at least in part, on the material identificationinformation of the one or more databases.
 14. The materialidentification system of claim 13, wherein the material identificationinformation includes at least one of function of scattering angleinformation, reciprocal lattice spacing information, and chemicalcomposition information.
 15. The material identification system of claim13, wherein the material identification information includes at leastone of structure data, chemistry data, morphology data, feature sizedata, and location data.
 16. The material identification system of claim1, further comprising a scanning transmission electron microscopeincluding the first sensor and the second sensor.
 17. A method ofidentifying a material sample, the method comprising: generating firstsensor data using a first sensor responsive to the material sample;generating second sensor data using a second sensor responsive to thematerial sample, the second sensor different from the first sensor; andapplying the first sensor data and the second sensor data to one or moreneural networks for: correlating the first sensor data to materialinformation stored in one or more databases to generate a firstpreliminary set of predictions of an identity of the material sample;correlating the second sensor data to material information stored in oneor more databases to generate a second preliminary set of predictions ofthe identity of the material sample; and narrowing the first preliminaryset of predictions responsive to the second preliminary set ofpredictions to generate a set of predictions of the identity of thematerial sample.
 18. The method of claim 17, further comprising rankingthe set of predictions of the identity of the material sample from aleast likely prediction to a most likely prediction, and selecting themost likely prediction of the identity of the material sample.
 19. Themethod of claim 17, wherein correlating the first sensor data and thesecond sensor data to material information stored in one or moredatabases comprises processing the first sensor data and the secondsensor data in parallel neural network stages of the one or more neuralnetworks.
 20. The method of claim 19, wherein the one or more neuralnetworks includes two chemistry stages configured to process diffractiondata, a diffraction stage configured to process chemistry data, and aclassification stage configured to process outputs of the two chemistrystages and the diffraction stage.